{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.rcParams['font.size'] = 16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import Ridge, LogisticRegression, SGDClassifier, SGDRegressor\n",
    "from sklearn.dummy import DummyClassifier, DummyRegressor\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "from sklearn.feature_extraction.text import CountVectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Big data sets: `SGDClassifier` and `SGDRegressor` (15 min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Something we haven't discussed in this course is huge data sets.\n",
    "- There could be a couple problems arising from huge data sets:\n",
    "\n",
    "1. The code is too slow.\n",
    "2. The dataset doesn't fit in memory - I can't even load it with `pd.read_csv`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Simplest strategy: subset your data for experimentation / hyperparameter tuning, then train your final model on the whole dataset (once)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- \"SGD\" (stochastic gradient descent) can help with both of these problems.\n",
    "- But we'll focus on using it to solve problem (1).\n",
    "- There is a fancy way to implement `fit` that can be a lot faster for big data sets.\n",
    "  - You can think of it as quickly finding \"approximately\" the best coefficients when calling `fit`.\n",
    "  - That is not quite true but it may be a useful way of thinking.\n",
    "  - Much more on this in CPSC 340 and much, much more on this in CPSC 440/540."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- SGD can be used in many contexts.\n",
    "- In sklearn, it's built in as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import SGDClassifier, SGDRegressor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `SGDRegressor` is basically equivalent to `Ridge`.\n",
    "- `SGDRegressor(loss='huber')` is basically equivalent to `HuberRegressor`.\n",
    "- `SGDClassifier(loss='log')` is basically equivalent to `LogisticRegression`, except the parameter is called `alpha` instead of `C` (like `Ridge`).\n",
    "- With other settings they are equivalent to other models, but this is good enough."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- For regular sized datasets, use the original functions, as these ones can be a bit more finicky. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load the [Sentiment140 dataset](http://help.sentiment140.com/home), which contains tweets labeled with sentiment associated with a brand, product, or topic. (I don't think we've looked at this dataset before - using it here because it's large.) You can download the data from [here](https://www.kaggle.com/ferno2/training1600000processednoemoticoncsv)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets_df = pd.read_csv('data/training.1600000.processed.noemoticon.csv', \n",
    "                        encoding = \"ISO-8859-1\",\n",
    "                        names=[\"label\",\"id\", \"date\", \"no_query\", \"name\", \"text\"])\n",
    "tweets_df['label'] = tweets_df['label'].map({0: 'neg', 4: 'pos'})\n",
    "tweets_df = tweets_df[tweets_df['label'].str.startswith(('pos','neg'))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets_df_train, tweets_df_test = train_test_split(tweets_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>id</th>\n",
       "      <th>date</th>\n",
       "      <th>no_query</th>\n",
       "      <th>name</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>843953</th>\n",
       "      <td>pos</td>\n",
       "      <td>1563972514</td>\n",
       "      <td>Mon Apr 20 01:07:37 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>JuanMoon</td>\n",
       "      <td>@JessLoebig this city is ridiculous lol... jk....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1134521</th>\n",
       "      <td>pos</td>\n",
       "      <td>1976350370</td>\n",
       "      <td>Sat May 30 17:08:07 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>MJJNews</td>\n",
       "      <td>@Lex_DH  Marvin Gaye, Stevie Wonder, anything ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>303843</th>\n",
       "      <td>neg</td>\n",
       "      <td>1999580980</td>\n",
       "      <td>Mon Jun 01 20:28:06 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>leeandradex3</td>\n",
       "      <td>@tommcfly of you in my computer, LOL please an...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>337900</th>\n",
       "      <td>neg</td>\n",
       "      <td>2014312351</td>\n",
       "      <td>Wed Jun 03 01:06:53 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>MizpahMijares</td>\n",
       "      <td>my right eye hurts with my contact lense,so i ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>215700</th>\n",
       "      <td>neg</td>\n",
       "      <td>1975543678</td>\n",
       "      <td>Sat May 30 15:22:56 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>KtShortcake</td>\n",
       "      <td>how do I load a profile pic, it keeps telling ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        label          id                          date  no_query  \\\n",
       "843953    pos  1563972514  Mon Apr 20 01:07:37 PDT 2009  NO_QUERY   \n",
       "1134521   pos  1976350370  Sat May 30 17:08:07 PDT 2009  NO_QUERY   \n",
       "303843    neg  1999580980  Mon Jun 01 20:28:06 PDT 2009  NO_QUERY   \n",
       "337900    neg  2014312351  Wed Jun 03 01:06:53 PDT 2009  NO_QUERY   \n",
       "215700    neg  1975543678  Sat May 30 15:22:56 PDT 2009  NO_QUERY   \n",
       "\n",
       "                  name                                               text  \n",
       "843953        JuanMoon  @JessLoebig this city is ridiculous lol... jk....  \n",
       "1134521        MJJNews  @Lex_DH  Marvin Gaye, Stevie Wonder, anything ...  \n",
       "303843    leeandradex3  @tommcfly of you in my computer, LOL please an...  \n",
       "337900   MizpahMijares  my right eye hurts with my contact lense,so i ...  \n",
       "215700     KtShortcake  how do I load a profile pic, it keeps telling ...  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets_df_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1200000, 6)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets_df_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Holy cow!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec = CountVectorizer(stop_words='english')\n",
    "\n",
    "X_train = vec.fit_transform(tweets_df_train['text']) \n",
    "y_train = tweets_df_train['label']\n",
    "\n",
    "X_test = vec.transform(tweets_df_test['text']) \n",
    "y_test = tweets_df_test['label']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1200000, 563780)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "scipy.sparse.csr.csr_matrix"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the fraction of elements that are nonzero. Having a sparse matrix really helps!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.2347450837797249e-05"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.nnz/np.prod(X_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's train a classifier. I'll use `time` instead of `%timeit` because I want to keep the output, and it gets lost with `%timeit`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/miniconda3/envs/cpsc330env/lib/python3.8/site-packages/sklearn/dummy.py:131: FutureWarning: The default value of strategy will change from stratified to prior in 0.24.\n",
      "  warnings.warn(\"The default value of strategy will change from \"\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.5002458333333333"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dc = DummyClassifier()\n",
    "dc.fit(X_train, y_train)\n",
    "dc.score(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2min 35s, sys: 5.26 s, total: 2min 40s\n",
      "Wall time: 24.5 s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/miniconda3/envs/cpsc330env/lib/python3.8/site-packages/sklearn/linear_model/_logistic.py:762: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LogisticRegression()"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%time lr.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.81016"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.score(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7765725"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([100], dtype=int32)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.n_iter_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_sgd = SGDClassifier(loss=\"log\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 5.03 s, sys: 85.6 ms, total: 5.11 s\n",
      "Wall time: 3.83 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "SGDClassifier(loss='log')"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%time lr_sgd.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_sgd.n_iter_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.769875"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_sgd.score(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.76577"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_sgd.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TODO\n",
    "\n",
    "think about the `C` and `alpha` hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `SGDClassifier` was about 10x faster than `LogisticRegression`, but the accuracy is slightly lower.\n",
    "- In fact, we can control the speed vs. _train_ accuracy tradeoff in both cases using the hyperparameters.\n",
    "  - The main ones are `max_iter` (higher is slower) and/or `tol` (lower is slower)\n",
    "  - (This is the same for both `LogisticRegression` and `SGDClassifier`)\n",
    "  - This is analogous to `n_iter` in `RandomizedSearchCV` !\n",
    "- In general, `LogisticRegression` will get slightly higher _train_ accuracy (may or may not correspond to better validation/test)\n",
    "- But in some cases your dataset is so big that `LogisticRegression` is not feasible, and then `SGDClassifier` can save the day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/miniconda3/envs/cpsc330env/lib/python3.8/site-packages/sklearn/utils/deprecation.py:143: FutureWarning: The sklearn.utils.testing module is  deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.utils. Anything that cannot be imported from sklearn.utils is now part of the private API.\n",
      "  warnings.warn(message, FutureWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 2 folds for each of 4 candidates, totalling 8 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done   3 out of   8 | elapsed:   15.4s remaining:   25.7s\n",
      "[Parallel(n_jobs=-1)]: Done   8 out of   8 | elapsed:   27.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=-1)]: Done   8 out of   8 | elapsed:   27.1s finished\n"
     ]
    }
   ],
   "source": [
    "# from https://scikit-learn.org/stable/auto_examples/linear_model/plot_sgd_early_stopping.html#sphx-glr-auto-examples-linear-model-plot-sgd-early-stopping-py\n",
    "\n",
    "from sklearn.utils.testing import ignore_warnings\n",
    "from sklearn.exceptions import ConvergenceWarning\n",
    "\n",
    "@ignore_warnings(category=ConvergenceWarning)\n",
    "def fit_grid_search_lr(iters):\n",
    "    grid_search_lr = GridSearchCV(LogisticRegression(), {\"max_iter\" : iters}, return_train_score=True, verbose=2, cv=2, n_jobs=-1)\n",
    "    # n_jobs=-1 might mess up the timing a bit but otherwise it takes too long\n",
    "    grid_search_lr.fit(X_train, y_train)\n",
    "    return grid_search_lr\n",
    "    \n",
    "grid_search_lr = fit_grid_search_lr([3, 10, 30, 100])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_fit_time</th>\n",
       "      <th>std_fit_time</th>\n",
       "      <th>mean_score_time</th>\n",
       "      <th>std_score_time</th>\n",
       "      <th>param_max_iter</th>\n",
       "      <th>params</th>\n",
       "      <th>split0_test_score</th>\n",
       "      <th>split1_test_score</th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>std_test_score</th>\n",
       "      <th>rank_test_score</th>\n",
       "      <th>split0_train_score</th>\n",
       "      <th>split1_train_score</th>\n",
       "      <th>mean_train_score</th>\n",
       "      <th>std_train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6.140643</td>\n",
       "      <td>0.085187</td>\n",
       "      <td>3.163987</td>\n",
       "      <td>0.017652</td>\n",
       "      <td>3</td>\n",
       "      <td>{'max_iter': 3}</td>\n",
       "      <td>0.675703</td>\n",
       "      <td>0.676132</td>\n",
       "      <td>0.675917</td>\n",
       "      <td>0.000214</td>\n",
       "      <td>4</td>\n",
       "      <td>0.676423</td>\n",
       "      <td>0.676437</td>\n",
       "      <td>0.676430</td>\n",
       "      <td>0.000007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7.696027</td>\n",
       "      <td>0.061743</td>\n",
       "      <td>3.257270</td>\n",
       "      <td>0.043842</td>\n",
       "      <td>10</td>\n",
       "      <td>{'max_iter': 10}</td>\n",
       "      <td>0.752872</td>\n",
       "      <td>0.752698</td>\n",
       "      <td>0.752785</td>\n",
       "      <td>0.000087</td>\n",
       "      <td>3</td>\n",
       "      <td>0.755837</td>\n",
       "      <td>0.755035</td>\n",
       "      <td>0.755436</td>\n",
       "      <td>0.000401</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>12.475981</td>\n",
       "      <td>0.124850</td>\n",
       "      <td>2.156903</td>\n",
       "      <td>0.074801</td>\n",
       "      <td>30</td>\n",
       "      <td>{'max_iter': 30}</td>\n",
       "      <td>0.769417</td>\n",
       "      <td>0.770663</td>\n",
       "      <td>0.770040</td>\n",
       "      <td>0.000623</td>\n",
       "      <td>2</td>\n",
       "      <td>0.781283</td>\n",
       "      <td>0.783087</td>\n",
       "      <td>0.782185</td>\n",
       "      <td>0.000902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>22.457827</td>\n",
       "      <td>0.176986</td>\n",
       "      <td>1.275458</td>\n",
       "      <td>0.003157</td>\n",
       "      <td>100</td>\n",
       "      <td>{'max_iter': 100}</td>\n",
       "      <td>0.772272</td>\n",
       "      <td>0.772438</td>\n",
       "      <td>0.772355</td>\n",
       "      <td>0.000083</td>\n",
       "      <td>1</td>\n",
       "      <td>0.825927</td>\n",
       "      <td>0.829435</td>\n",
       "      <td>0.827681</td>\n",
       "      <td>0.001754</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean_fit_time  std_fit_time  mean_score_time  std_score_time  \\\n",
       "0       6.140643      0.085187         3.163987        0.017652   \n",
       "1       7.696027      0.061743         3.257270        0.043842   \n",
       "2      12.475981      0.124850         2.156903        0.074801   \n",
       "3      22.457827      0.176986         1.275458        0.003157   \n",
       "\n",
       "  param_max_iter             params  split0_test_score  split1_test_score  \\\n",
       "0              3    {'max_iter': 3}           0.675703           0.676132   \n",
       "1             10   {'max_iter': 10}           0.752872           0.752698   \n",
       "2             30   {'max_iter': 30}           0.769417           0.770663   \n",
       "3            100  {'max_iter': 100}           0.772272           0.772438   \n",
       "\n",
       "   mean_test_score  std_test_score  rank_test_score  split0_train_score  \\\n",
       "0         0.675917        0.000214                4            0.676423   \n",
       "1         0.752785        0.000087                3            0.755837   \n",
       "2         0.770040        0.000623                2            0.781283   \n",
       "3         0.772355        0.000083                1            0.825927   \n",
       "\n",
       "   split1_train_score  mean_train_score  std_train_score  \n",
       "0            0.676437          0.676430         0.000007  \n",
       "1            0.755035          0.755436         0.000401  \n",
       "2            0.783087          0.782185         0.000902  \n",
       "3            0.829435          0.827681         0.001754  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_lr = pd.DataFrame(grid_search_lr.cv_results_)\n",
    "results_lr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 2 folds for each of 3 candidates, totalling 6 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done   3 out of   6 | elapsed:    9.6s remaining:    9.6s\n",
      "[Parallel(n_jobs=-1)]: Done   6 out of   6 | elapsed:   15.6s finished\n"
     ]
    }
   ],
   "source": [
    "grid_search_sgd = GridSearchCV(SGDClassifier(loss=\"log\", max_iter=100_000), {\"tol\" : [1e-3, 1e-4, 1e-5]}, return_train_score=True, verbose=2, cv=2, n_jobs=-1)\n",
    "grid_search_sgd.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_fit_time</th>\n",
       "      <th>std_fit_time</th>\n",
       "      <th>mean_score_time</th>\n",
       "      <th>std_score_time</th>\n",
       "      <th>param_tol</th>\n",
       "      <th>params</th>\n",
       "      <th>split0_test_score</th>\n",
       "      <th>split1_test_score</th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>std_test_score</th>\n",
       "      <th>rank_test_score</th>\n",
       "      <th>split0_train_score</th>\n",
       "      <th>split1_train_score</th>\n",
       "      <th>mean_train_score</th>\n",
       "      <th>std_train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3.670437</td>\n",
       "      <td>0.092043</td>\n",
       "      <td>2.178412</td>\n",
       "      <td>0.024795</td>\n",
       "      <td>0.001</td>\n",
       "      <td>{'tol': 0.001}</td>\n",
       "      <td>0.764978</td>\n",
       "      <td>0.766073</td>\n",
       "      <td>0.765526</td>\n",
       "      <td>0.000548</td>\n",
       "      <td>1</td>\n",
       "      <td>0.772828</td>\n",
       "      <td>0.772725</td>\n",
       "      <td>0.772777</td>\n",
       "      <td>0.000052</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5.325876</td>\n",
       "      <td>0.051982</td>\n",
       "      <td>2.072479</td>\n",
       "      <td>0.040767</td>\n",
       "      <td>0.0001</td>\n",
       "      <td>{'tol': 0.0001}</td>\n",
       "      <td>0.764853</td>\n",
       "      <td>0.766190</td>\n",
       "      <td>0.765522</td>\n",
       "      <td>0.000668</td>\n",
       "      <td>2</td>\n",
       "      <td>0.772860</td>\n",
       "      <td>0.772322</td>\n",
       "      <td>0.772591</td>\n",
       "      <td>0.000269</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>12.365952</td>\n",
       "      <td>0.110125</td>\n",
       "      <td>1.278813</td>\n",
       "      <td>0.016750</td>\n",
       "      <td>1e-05</td>\n",
       "      <td>{'tol': 1e-05}</td>\n",
       "      <td>0.765028</td>\n",
       "      <td>0.765907</td>\n",
       "      <td>0.765467</td>\n",
       "      <td>0.000439</td>\n",
       "      <td>3</td>\n",
       "      <td>0.773053</td>\n",
       "      <td>0.772467</td>\n",
       "      <td>0.772760</td>\n",
       "      <td>0.000293</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean_fit_time  std_fit_time  mean_score_time  std_score_time param_tol  \\\n",
       "0       3.670437      0.092043         2.178412        0.024795     0.001   \n",
       "1       5.325876      0.051982         2.072479        0.040767    0.0001   \n",
       "2      12.365952      0.110125         1.278813        0.016750     1e-05   \n",
       "\n",
       "            params  split0_test_score  split1_test_score  mean_test_score  \\\n",
       "0   {'tol': 0.001}           0.764978           0.766073         0.765526   \n",
       "1  {'tol': 0.0001}           0.764853           0.766190         0.765522   \n",
       "2   {'tol': 1e-05}           0.765028           0.765907         0.765467   \n",
       "\n",
       "   std_test_score  rank_test_score  split0_train_score  split1_train_score  \\\n",
       "0        0.000548                1            0.772828            0.772725   \n",
       "1        0.000668                2            0.772860            0.772322   \n",
       "2        0.000439                3            0.773053            0.772467   \n",
       "\n",
       "   mean_train_score  std_train_score  \n",
       "0          0.772777         0.000052  \n",
       "1          0.772591         0.000269  \n",
       "2          0.772760         0.000293  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_sgd = pd.DataFrame(grid_search_sgd.cv_results_)\n",
    "results_sgd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZoAAAEQCAYAAACJLbLdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAABAEklEQVR4nO3deXxU5dXA8d/JniAJS4CEBEjCvilGVFAERBFBhAJaRQG1at1qaX1rlVoL4trWou3r8qpoEVy6EcEFF1BRtCxC2AlrEjAJSxbWkHXyvH/cCUzCJJkkM5lJ5nw/n/nM5N7n3jkzDHPmuc8mxhiUUkopTwnwdgBKKaVaNk00SimlPEoTjVJKKY/SRKOUUsqjNNEopZTyqCBvB+CLoqOjTUJCgrfDUEqpZmXDhg15xpgO1bdronEiISGB9evXezsMpZRqVkRkv7PteulMKaWUR2miUUop5VGaaJRSSnmUJhqllFIepYlGKaWUR2miUUop5VGaaJRSSnmUJhqllPJztgrDt7tz+dNnOz1yfh2wqZRSfmrXoZOkpGaxZFM2h0+UEBkWxO2XJ9CxdZhbn0cTjVJK+ZHckyUs3ZTNBxuz2Z5zgqAAYWTvDsy+Pp5RfToSFhzo9ufURKOUUi1ccZmN5TsOk5Kaxbd78rBVGM6Pj2LO9f24/oLOtD8v1KPPr4lGKaVaoIoKww+ZBaSkZrNs60FOlpQTGxXGz4cnMfnCOHp2at1ksWiiUUqpFiQjr5APUrNI2ZhN1tEiIkICGTsglinJcVya1J7AAGnymDTRKKVUM3fsdCkfbTnIB6lZpB44hggM6xHN/1zTizH9Y4gI8e5XvSYapZRqhkrLK1i56wgpqdl8tfMIpbYKenU6j1lj+zBxUBwxUe7tOdYYmmiUUqqZMMawOes4KalZfLQ5h6Ony4g+L4RpQ7oxOTmO/p0jEWn6S2N10USjlFI+LvtYEUs2ZrM4NYv03EJCggK4pl8npiTHM6xnNMGBvj32XhONUkr5oJPFZXy67RApqVmsSS8A4JLEdvz8iiTGDowlKjzYyxG6ThONUkr5iHJbBd/vyyclNYvPtx+iuKyChPYRPDS6F5MujKNLuwhvh9ggmmiUUsrL0g6eICU1i6WbcjhysoSo8GCmJMczOTme5K5tfLLdpT400SillBccOVnMh5tyWJyaTdpBayqYK/t0ZEpyHFf26UhokPungvEWTTRKKdVEikptfLHjECmp2azak0uFgQu6tGHuxP6MP78z7VqFeDtEj9BEo5RSHlRRYViXWUBKahbLth7iVEk5naPCuG9kdyZdGE+Pjud5O0SP00SjlFIesC/3FB+kWrMkZx8rolVIIOMGxjI5OZ5LE9sR4IWpYLxFE41SSrnJ0cJSPt5itbts+vEYAQLDenbgt9f25pp+MYSHtJx2l/rQRKOUUo1QUm7j6525pKRm8fWuI5TZDH1iWvPYuL5MHNSZjpG+MxWMt2iiUUqpejLGsOnHY6SkZvPRlhyOnS4j+rxQbhuawOTkePp1jvR2iD5FE41SSrnox4LTLNlotbuk5xUSGhTAmP4xTEqO44oe0QT5+FQw3qKJRimlanGyuIxPtx5icWoWazOsqWAuTWzHvSO6M3ZgDK3Dms9UMN6iiUYppaopt1Wwam8eKanZfLH9ECXlFSRFt+I31/Ri4qDmOxWMt2iiUUopux051lQwSzblkHeqhDYRwfx0cBcmJ8cxqEvznwrGWzTRKKX82uETxSzdlE1KajY7D50kOFAY1acjk5PjubJ3R0KCtN2lsTTRKKX8TuVUMItTs/nOPhXMoC5teNI+FUzbFjoVjLdoolFK+YWKCsOajHxSUrP5dOtBCkttxLUJ54ErezDpwjiSOrT8qWC8RRONUqpF23vkFB9szGLJxhyyjxVxXmgQ151vTQVzSYJ/TQXjLZpolFItTkFhKR9tziElNYvNWccJEBjeqwOPjO3D6L6d/HYqGG/RRKOUahGsqWCOsDg1m693HqG8wtAvNpLfX9eXCYM607G1TgXjLZpolFLNljGG1APHSEnN4uMtBzleVEaH1qH8bFgiky6Mo2+sTgXjCzTRKKWanR8LTvPBxmxSUrPIzD9NWHAA1/aPYVJyPJd3b69TwfiYJk80ItIFeAEYDQiwAviVMeaAC8d2BZ4ErgSigSzgX8CzxphCh3IBwCPAPUAMsAuYa4xZ7N5Xo5RqKieKy1i25SApqdmsyyxABIYktueBK3swdmAs54Xq72Zf1aT/MiISAXwFlAC3AQZ4CvhaRM53TBZOjm2FlZSCgceBA8DFwBNAT+Amh+JPAr8BHgM2ADcD/xaR8caYZe5+XUopzyi3VbBqTx6LU7NYvuOwNRVMh1Y8PKY3P7kwjrg24d4OUbmgqX8C3A0kAb2NMXsBRGQLsAer9jGvlmMvx0ooY4wxX9i3fS0i7YDfiEiEMea0iHTESjLPGWOedyjXA3gO0ESjlA8zxrA95wQpqdl8uDmbvFOltI0I5uaLuzA5OZ7z46N0KphmpqkTzQRgTWWSATDGZIjI98BEak80lUN1T1TbfgwIwLoMBzDGXvadauXeAd4SkURjTEbDwldKecqh42engtl1+CQhgQFc1deaCmZErw46FUwz1tSJpj+w1Mn27cCNdRy7Aqvm80cRuQ/r0tklwEzg/xwuu/XHujS3t9rx2+33/QBNNEr5gNOl5Xy+/RApqdl8tzcPYyC5axue+skAxp8fS5sInQqmJWjqRNMOOOpkewHQtrYDjTHFIjIMWMzZpAEwH/hFtec4ZowxTp6jcv85ROTnwM8BunbtWlsoSqlGsFUY1qTbp4LZdpDTpTbi24bz4KieTLowjsToVt4OUbmZN7ppVE8AcPayV41EJAz4J9ARmM7ZGs0fgHLgPodz1fs5jDGvA68DDB482NnxSqlG2HvkJItTs1myMZuDx4tpHRrEhAs6Mzk5nsHd2upUMC1YUyeaozivUbTFeU3H0Z3ASKCHMWaffdu3InIceF1E/s8Ysxl77UhEpFqtprLGVIBSqknknyqxpoLZmM2WrOMEBggjenXgsev6cnXfToQF61Qw/qCpE812rDaU6voBO+o4diBw1CHJVFpnv+8LbLY/RyjQnartNP3s93U9j1KqEYrLbHy18wgpqVms3JVLeYVhQFwkj4/vx4QLOtOhdai3Q1RNrKkTzYfA8yKSZIxJBxCRBKyuy4/WcewhrJpKD8dea8Cl9vts+/1nQClwK9YYm0rTgG3a40wp97OmgjnK4tRsPt6cw4nicjpFhnLnFYlMvjCe3jGtvR2i8iKXE419wOSdwHCgPfBzY8weEbkZ2GSM2enCad7AarhfKiK/x2pLeRL4EXjN4bm6AfuwRvPPtW9eADwELBORp7HaaAZjDd7cAHwPYIw5IiIvALNE5CSQijWYcxRWF2qllJscyD9NysYsPtiYzf7804QHB3LtgBgmJ8dxWfdoArXdReFiorFPG7MSiAd2AgOAyp8oVwJXA3fVdR5jTKGIjMKagmYRVgP9l1hT0JxyfEogEGt8TOWxmSIyBJiDNZtANFaCeh142hhT4XD8Y8AprK7PlVPQ/NQY85Err1cpVbPjRWUs23qQlNQsfsg8ighc1r09vxzVk2sHxNBKp4JR1bj6ifgL1tiUnkAO1qWpSt9gffm7xD6n2ZQ6ymTipJeYMWYH8FMXnsOGlYyecjUupVTNymwVfLs7l5TUbJanHaa0vIIeHc/jt9f25ieD4uisU8GoWriaaEZjXSo7ICLVu4lkA3HuDUsp5W2VU8EsTs3iw0055BeW0q5VCLdc0pUpyfEMiIvUqWCUS1xNNCHAyRr2RQFl7glHKeVtB48XsWSjtTrlniOnCAkMYHS/Tky6MI4RvTsQrFPwq3pyNdFswbrc9ZmTfWOxGuOVUs1UYcnZqWC+32dNBTO4W1uemTSQ6wbGEhUR7O0QVTPmaqL5M/AfezX5Pfu2fiIyEasn2gQPxKaU8iBbhWH1vnxSUrP4bPshTpfa6NougplXWVPBdGuvU8Eo93Ap0RhjUkTkfqxp9n9m37wQ63LaL4wxzmo6SikftPvwSVLsU8EcOlFM67AgJg6KY0pyHBd1a6vtLsrtXO3eHAX8HatL8lCs+cbygf8aY2pqu1FK+Yi8UyV8uCmHlI1ZbMs+QVCAMLJ3Bx4f34+r+nbUqWCUR9WZaEQkCCupTLKPQ1nh8aiUUo1WXGZjRdphPkjNZuXuXGwVhoFxUcy+vh/XX9CZ6PN0KhjVNOpMNMaYchE5DNiaIB6lVCMYY1i//ygpqVl8vOUgJ4vLiYkM4+fDk5h8YRw9O+lUMKrpudoZ4B2skf+6DLJSPmh/fiEpqdl8sDGbAwWniQixpoKZkhzPkKT2OhWM8ipXE00mcIuI/IC1QuZBqq35Yox5y72hKaVqc/x0GR9vzSElNZsN+62pYIb1iObXo3sypn8MESE6FYzyDa5+El+238cBFznZbwBNNEp5WJmtgpW7cvlgYxYrdhyh1FZBr07n8ejYPkwc1JnYKJ0KRvkeVxNNokejUErVyBjD1uzjpKRm8+HmHAoKS4k+L4RpQ7oxOTmO/p11Khjl21wdR7Pf04EoparKOVbEkk3ZpKRms/fIKUKCrKlgpiTHcUVPnQpGNR/1uogrIgOAEVjLMecD3xpjtnkiMKX80amScj7bdoiU1CxWp+djDFyS0I7nJg9k7MBYosJ1KhjV/Lg6YDMIa+GxqVSdvt+IyHvA7fap+ZVS9VRRYfh+Xx4pqdl8tu0QRWU2urWP4FdX9WLShXF0bR/h7RCVahRXazSzsdaB+QNWV+dDWAuKTbPvS7ffK6VcVFxmY8nGbOZ/l8HeI6eIDAticnIck5PjSe7aRttdVIvhaqKZBjxpjHnaYdt+4Gn7+jR3oIlGKZcUFJayaPV+Fq3JJO9UKf07R/LiTYMYOzCG0CCdCka1PK4mms7A6hr2/Rdr6WSlVC3Sc0/x5ncZLE7NorisglF9OnLXFYkMTWqvtRfVormaaHKAy3E+z9ll9v1KqWqMMazLKOCNVRl8ufMwwYEBTL4wjruuSKRHR50ORvkHVxPNu8BjIlJhf3wQq43mZqzazB89E55SzVO5rYJPtx1i/qp0Nmcdp12rEB4c1ZPpQ7rRobVOZqn8i6uJZg6QBDxhf1xJgPft25Xye6dKyvnHugP8/ftMso8VkRTdiqcnDWBKcrxOxa/8lqsDNsux5jp7GhiONY6mAPjGGLPDg/Ep1SzkHCtiwX8zeX/tAU6WlHNJYjvmTOjPVX06EqATWio/V68Bm8aY7cB2D8WiVLOzLfs481el8/GWgxhg3MBY7r4ikfPj23g7NKV8hqsDNu8Auhlj5jjZNwfIMMa87d7QlPJNFRWGlbuP8Ma3GaxOz+e80CBuuyyBOy5PIL6tDq5UqjpXazQzgTdr2HcE+BWgiUa1aNUHWMZGhfG7cX24+ZKuRIbp1DBK1cTVRNODmi+ZpQHd3ROOUr7H2QDLv948iHEDY3ViS6Vc4GqiKQeia9jXwU2xKOVT9lUOsNyQRUm5DrBUqqHEGFN3IZEvgCBjzCgn+74CjDHmKg/E5xWDBw8269evr/dxP1nyE9KPpxMogYgIARJw9kYAAQHWvYhUKRMogQjVyjvcKvdVPy9YAwIdGfvCp5Xbz/xduSCqca1cfc/r7HPU6HNX+9vV89b4Wl08b5nNRnF5BWW2CgBCAoXQ4ADO5JYGnremcjP6zeBXF/0KpepkDNjKoKIcKsrAVnlfua3c/tj+d5X9NofHlcc6OX7I/RDQsK74IrLBGDO4+nZXazRPAytEZC0wH8jGWm3zLiAZGN2gqFqYn/b+KfnF+RhjqDAVZ29UVP3bfjMYbBU2DFZ5m7HVeqzjPpuxWb+qBcQ+oXbl/dm7qtsrf4XXt1y9z+uw7czf9T13DeVcPq+TmGo7rwH2558m7eAJCk6VEhYcSN9OrekTE0l4SGCDz+vKe35RJ2eL1qpGMabaF2v1L2Gbwxeuk/1VvoTLHb64yxrxZV7TeWp77mr7m2KS/IvvggD3rtTq6jiab0TkBuBF4DWHXZnAFGPMSrdG1Uzd0vcWb4eg6ulkcRn//OHHKgMs/3BFon8NsKyocO2Xbp1fsvX5Mnf8kq3ry9zWsNiakgRCYDAEBFm3wGAIsP8dGGQ9Pmd/EASH27cFVysX6PDY/nfl48Agh2OCq57P8TznPHf181R/Hvt5gsLc/va4PI7GGLMUWCoivYH2QJ4xZrfbI1KqCfjMAEtjIH8vZK6C0sLav4Sd/WKu60vYlUsq1S71eZa4/iXs+OUZEuHil2dtX8KuJoAaznPOfoftAdoppDb1GrAJYIzZVflYRNobY/LdG5JSnrMt+zhvrErnE28OsCwrhszvYM8X1u1ohvNyVb7onPzydPYLNigEAlo14MuzHl/C5/wqriVBnHMeP6klqipcHbB5N9DGGPNn+98DgU+BWBHZCIw3xhzyXJhKNZxPDLA8dsCeWJZD+jdQXgRB4ZA4HC77BXQfBRHRVb/MtWebaiFcrdE8CLzu8Pc84BjWrM2/BOYCP3drZEo1UnGZjQ82ZjN/VTr7cgubdoClrQwOrDmbXHLTrO1tukHydOg5BhIut67RK9XCuZpougI7AUQkChgB/MQYs0xE8oFnPRSfUvWWf6qEd9YcYOHqTPILm3CA5cnDsHe5lVz2fQ0lJ6zaSbfL7MnlGmjfQ2sqyu+4mmgCgQr742FYrYcr7X//CHR0b1hK1V+TD7CssEF26tm2loObrO2tY6H/T6xaS9IICNUFzpR/czXR7AGuA77CWuzsv8aY0/Z9nbGWDPB7+fPnU57XgL4RDf0SrO24GnbV+IVbawwNOKbG56lneWqJGcFgyD5WzMb9R0nPKyQ4QHgiNpILu7ah3ckMWLaWvMbEVf2YsiIoSIf8fUjBPig7bR0cFQ/tx0P7nhDRCXIEcrLgm3frfKLwQRfQasiQWgJQqnlzNdE8DywSkduAtsCNDvuuBLa4O7Dm6MRnn1Oanl6vYxrcsbS2GR1q2lff7bXsqzXuJnh+x+1hwFD7DYA0q/qdd+5RbhYERNofHwdS7bf6aXfnzzTRqBbN1QGb74nIAeBS4AdjzLcOuw8DH3oiuOYm8T//9nYILZ6zAZZ3ujjAssbplqpvLzkJ6SutRvw9K+DkQWt77AXQc7R1SSz2AudddRuSNHUMhmrh6jNg8zvgOyfbZ7s1IqWccMcAy1rbafL2nG1r2f9fazBjaCR0v9JKLD2uhtad3PRqlPIv9R6wqVRT8tgAy7KiaoMmM63tHfrCkPug1xjocqk1rkUp1SiaaJTPqagwfL3rCG+sSmdNeoH7Blge3X92XEvGt2cHTSaNgMt+aV0Wa9PVfS9EKQVoolE+xO0DLG1lcGC1w6DJndb2tgmQPAN6XQPdhkGw+ycRVEqdpYlGeV3+qRIWrdnPotX7Gz/A8uQheyO+fdBk6Ulr0GTC5VZy6TkG2nfXQZNKNSFNNMpr9uWeYv6qDFJSGzHAssIG2RscBk1utra37gwDJlttLYnDddCkUl7k6qSa3wP/B/zLGFPi2ZBUS2aMYW1GAfNXpbMi7QghQQFMSY7jzmGJ9OjoYjI4XQB7v7QSy94VUFQAEmA13l/1B6vW0qm/1lqU8hGu1mjKgLeBF0XkbeB1Y8xOz4WlWpoyWwXLth5k/qoMtmYfp12rEH55VU9mDO1G9HmhtR9sDBzacratJesHMBUQ0d6aP6zXNZB0JUS0a5oXo5SqF1cHbI60L3h2DzADmCkiq4BXgRRjTJkHY1TNmLMBlk9PGlD3AMuSk1YbS2VyOWVfhaLzhTD8YavW0vlCHeyoVDNQnwGbu4CHRGQW8FOsZQHeA/JE5O9YtZz6zb+iWqycY0X8/fsM/rHuR06WlHNpYjuemNCfUTUNsDTGPmjyc/ugydX2QZNR1qDJXvZBk+fp/K1KNTcNWWGzBGves+1Y69IMB34L/EZEPgAe1EXQ/Ff2sSL+9NlOPt5iTdtS6wDLsiLIWHW2If/Yfmt7x34w9H6r1tLlEh00qVQzV69EIyLhwFTgXuAirDVqZgL/Bq4H5gDvAle5NUrVbDz18Q6+3nWE22saYHk082z344xvobwYgiMgcQRcPtNqc2nTxSuxK6U8w9VeZwOx2mduBVoBS4FHjDFfOxR7Q0QOYSUd5YcqKgxr0vO5bmBnHh/fz9pYXlp10GTeLmt720S46HYrsXS7XAdNKtWCuVqj2QzkAC9itcUcrKHcXmB1bScSkS7AC8BorMU5VgC/MsYcqOO4OUBNE3iWGGPCHMpmAt2clJtkjFlS2/Oohtube4qjp8sY0bkcUhfaB02utAZNBoZYCeWi2632lvbdvR2uUqqJuJpobgSWGGNstRUyxqRhrU/jlIhEYC2eVgLchrWsyVPA1yJyvjGmsJbTzwc+q7atlX2bs2UKPse6lOdoV23xq8ZZm57P7YGfMWHFQmtDZBwMnGLVWhJHQOh53g1QKeUVriaaD7HWlzonEYhIK6DUxS7OdwNJQG9jzF778VuwVvC8B6tzgVPGmCwgq9pzT7e/hredHJJnjFnjQkzKTdZkFPDrkG8wMRcgE1/RQZNKKQBcHYQwH3ijhn2v2W+umACsqUwyAMaYDOB7YKKL53B0G9bCa5834FjlRsYYdqVn0sPsR/pOgJgBmmSUUoDrieZKrA4AznyI673M+gPbnGzfDvRz8RwAiEi8Pa53jTHlTopcLyKnRaRERNaIyE/qc35VPxl5hfQ4bZ9nLOEK7wajlPIpriaajsCRGvblAq4uPdgOOOpkewHQ1sVzVJqOFb+zy2YfAQ8CY7B6yhUDH4jItJpOJiI/F5H1IrI+Nze3nqGotRkFDAnYQUVQuDViXyml7FxNNEeAgTXsGwjk1+M5nS2c3pBrLDOAjcaYLec8gTEPGmMWGmNWGWP+g1XjWg88W2NQxrxujBlsjBncoUOHBoTj39ZlFHBFUBrSbSgEhXg7HKWUD3G1M8DHwOMistLxi90+vuYx4AMXz3MUq1ZTXVuc13ScEpFLgD7Ar1wpb4yxici/gT+KSGwt3bNVAxhj2L1vH935ERLu8HY4ykOKi4vJzc2luLiY8nJnV6tVSxQUFERYWBgdOnQgLKxh491cTTR/wBr3skFEfsDq/RUHXAJkAL938TzbsdppqusH7HDxHGB1AijHmmvNVZW1Jmc1KtUIWUeL6HpqM4QACcO9HY7ygOPHj3P48GE6dOhATEwMQUFBrq8ZpJotYwzl5eWcOnWKAwcO0KlTJ6Kioup9HpcunRlj8oCLsS49CTDIfv80cLF9vys+BIaISFLlBhFJAC7H+ViYc4hICHAzsMwY41JjiogEYY0FOqDzsLnf2owChgbsoCIoAjoP8nY4ygPy8vKIj4+nbdu2BAcHa5LxEyJCcHAwbdu2JT4+nvz8+rSSnFWf2ZuPYdVs/tCgZ7K8AfwCWCoiv8eqXTwJ/IhDF2kR6QbsA+YaY+ZWO8d4rMtvzjoBICJTsbpKL7OftxPwANbcbFMbEbuqwdr0fO4NSkO6XaYTYLZQpaWlhIeHezsM5UXh4eGUlDRs3csmXcrZGFMoIqOwpqBZhFUr+hJrCppTDkUFCMR5jes2rF5qH9fwNBlYveT+jJWQTgM/ANcaY3S8jQfsTk+nO1mQeJe3Q1EepLUY/9aYf3+XE42IDADuBHpjzRLgyBhjXBpLY5/TbEodZTKpoSeaMabWgZ322QBGuRKLaryDx4uIP55qb5/R8TNKqXO5OnvzpcA3QCbQE9iC1VOsK1bHgL01HqxatHUZBQwN2I4tuBWBsYO8HY5Syge5Oo7mGSAFq8eYAHcaYxKAq7EucT3lkeiUz1uTXsBlgTsJ6HYZBDbplVilGmXOnDmNuhyUmZnJnDlzSE93/8LCK1euRERYuXKl28/tDa4mmvOBdzjbNTgQwBjzFVaSqXEgpGrZ9uzbS5JkI4l62Uz5l8zMTJ544gmPJJrk5GRWr15NcnKy28/tDa7+BA0GCo0xFSJSAMQ67NsFDHB7ZMrn5Z4sIfboenv7zDBvh6OU29lsNowxBAU1rrZujKGsrIyQENdmzYiMjGTIkCGNek5f4mqNZh/WAE2w2md+JiIBIhIA3AHo2BQ/tC6jgCEBadiCW0PMBd4OR6lGExEee+wxnnvuORITEwkJCWHr1q3nlFu5ciVXXmktvTV69GhEpMqlroSEBKZNm8Zbb71Fnz59CAkJ4ZNPPgFg9uzZJCcnExUVRXR0NKNGjWLNmjXnnL/6pbORI0cybNgwVqxYQXJyMhEREQwYMIAlS5Z45L1wp/pMQTMSayT+M8AnwAnABpwH/NITwSnftjYjnzsCdyAJQ7V9RrUYCxYsICkpieeff55WrVrRuXPnc8okJyfz8ssv88ADD/C3v/2Niy++GIB+/c5OQv/111+zadMmZs+eTceOHUlISAAgOzubX//618THx1NYWMg777zD8OHDWb9+Peeff36tse3bt4+ZM2cya9YsoqOj+ctf/sINN9zAzp076dGjh/veBDdz6dvBGDPb4fEKERmC1UU5AvjMGPOFh+JTPmzvvj0kykFIuN/boSgveeKj7ezIOeHVGPp1jmT29c5mtmoYYwxffPFFrQNUIyMjzySVvn37Or3MdfToUTZs2EBMTEyV7fPnzz/z2Gazce2119K/f3/efPNN/vrXv9YaW15eHt9++y09e/YErIQXGxvLv/71L373u9+5/BqbWp2JRkSCgXHAFvsiZRhjNgIbPRyb8mFHC0uJzrO3z2hHANWCXHvttW6ZBWHIkCHnJBmAFStW8PTTT7NlyxYKCgrObE9MTKzznD179jyTZAA6duxIx44dOXDgQKPj9aQ6E40xpkxE/gVcizXqXinWZRYwJGA75cGtCYqpvbqvWi531iR8RWxsbN2FGnie1NRUxo0bx5gxY3jzzTeJjY0lMDCQu+66i+Li4jrP2a7duZPfh4aGunSsN7l6YT0da1oXpQCrI8C0wDQk4XIICPR2OEq5jbum2nF2nsWLFxMUFERKSgrBwWfnBTx69Cht2rRxy/P6Ild7nf0JeExEdEUwBcDevbtIlEME6mUz5adCQ0MBKCoqcvmY06dPExgYWCUJffXVVz5/6auxXK3RjMKaoDJDRNYAB6m6rosxxtzm7uCUbzpRXEa73HXW6CpNNMpP9erVi6CgIN566y3atWtHaGgovXv3pnXr1jUec+211/Liiy9y++23c8cdd7B7926efPJJ4uLiajymJXC1RjMMKANyge72v6+odlN+YkPmUS6VHZSHREInHaur/FP79u156aWX2Lx5MyNGjODiiy9mw4YNtR4zZswY/va3v/H9998zfvx43nrrLRYuXOjTXZPdQYzRBSerGzx4sFm/fr23w/BZz36axi1rJhLfK5nAW//h7XBUE0hLS6Nv377eDkN5WV2fAxHZYIwZXH27jrJT9bZvzy66yWFI0mWblVJ1c3WZgK51lbGvM6NauMKSctocWWt9cnR+M6WUC1yt0WRStfHfGe3j6gdSDxzlYnZQFhJFsLbPKKVc4Gqi+RnnJpr2wHVAEvCkO4NSvmttegE/DdxhHz/jal8SpZQ/c3WuswU17JonIouwko3yA+l70+gqRyBphLdDUUo1E+74SfoOVo1HtXDFZTZaH7RPZ67tM0opF7kj0XQEwtxwHuXjNh44xmC2UxrSBjr2q7O8UkqB673OnPVjDcFaWXMWsMqdQSnftC6jgMmBaUjCMG2fUUq5zNXOACs5tzNA5WQ93wD3uSsg5bv27dlOF8mF7jp+RinlOlcTzZVOthUD+40xuoyzHygtr6DVwdXWxdYEnXFIKeU6l65/GGO+cXJbq0nGf2zNPsZgs53SkLbQoY+3w1Gq0ebMmeO2JQFU7VxKNCIyRER+WsO+G0XkUveGpXzNmn35XBqQZvU20/YZpVQ9uPqN8SxQ01J6fe37VQuWvncH8ZJHSA8dP6P8h81mo7y83NthNHuuJpoLgDU17FsH6Fq+LVi5rYLwrP9af+j4GdWCiQiPPfYYzz33HImJiYSEhLB169YayxcWFvLoo4/SvXt3QkNDiYmJYcqUKRw+fJh169YhInz00UfnHHfffffRoUMHysrKPPlyfIarnQHCqDkpBQKt3BOO8kXbc05wodlGSUg7QrV9RrVwCxYsICkpieeff55WrVrRuXNnp+VKS0sZPXo0mzZtYtasWQwZMoTjx4/z+eefc/ToUS655BJ69+7NokWLuP7666sc969//YtbbrmlynLOLZmriSYNmAB84mTfBGCX2yJSPmdteh7jA3ZgEoaBNp4qR58+Codq/sXfJGIGwtjn3HY6YwxffPEF4eHhtZZ75513WL16NUuXLmXChAlntt9www1nHk+fPp2nnnqK48ePExUVBcCyZcsoKChg+vTpbovZ17l66ez/gLtF5M8i0ktEIkSkp4j8GbgTeMVzISpvS9+9nc5SQFhPbZ9RLd+1115bZ5IB+OKLL4iJiamSZKqbNm0aJSUl/Pvf/z6zbdGiRfTu3ZtLLrnELfE2B65OqvmGiPQGfg085LgLeMEY87onglPeZ6swhGZ/b/2h42dUdW6sSfiK2NhYl8rl5+cTFxdXa5lu3boxfPhwFi1axF133cWxY8f45JNPePzxx90RarPh8gqbxpjfiMirwNVYSwTkASuMMemeCk55385DJxhk20ZxRHvCont5OxylPM7VsTXR0dFs27atznLTp0/n7rvvZv/+/Xz++eeUlpZy6623NjbMZqVeAyKMMfuMMa8ZY54xxryuSablW7svnyEBaVR00/YZpRxdc801HDp0yGmvMkc33ngjYWFhvPvuuyxatIjhw4eTkJDQNEH6CFcHbN4hInNq2DdHRG5za1TKZ2Tu3kqsFBCh7TNKVTFt2jSGDh3K1KlTefrpp1mxYgUffPAB9957Lzt37jxTLjIykgkTJvDyyy/z/fff+1UngEqu1mhmAvk17DsC/Mot0SifYowhOMvePpOoE2kq5Sg4OJgvvviC++67j9dff51x48Zx//33k5eXR7t27aqUnT59Ojk5OYSGhlbpleYvxJjqkzI7KSRyCphgjPnKyb4rgaXGmEgPxOcVgwcPNuvXr/d2GF63+/BJ0l76KddE7Cb80b166cyPpaWl0bdvX2+Hobysrs+BiGwwxgyuvt3VGk05EF3Dvg4unkM1M2vT8xkSsEPbZ5RSjeJqolkH3FvDvnuBH9wTjvIlGbs20UmOEdFrpLdDUUo1Y652b34aWCEia4H5QDYQB9wFJAOjPROe8hZjDEEHrPnNRNtnlFKN4OqAzW9E5AbgReA1h12ZwBRjzEq3R6a8KjP/NAPKtnA6oiMR7ZK8HY5Sqhmrz4DNpcBS+wwB7YE8Y8xuj0WmvGrtvjyuCtiBresobZ9RSjWKy4mmkjFGJ9D0Axm7NtFBjmP6OFvFWymlXFevRCMiFwC9sZYNqMIYs9BdQSnvCzpgjZ8RXX9GKdVILiUaEWmDtUTAkMpN9nvHQTiaaFqIHwtO07dkM4URnWil7TNKqUZytXvzM1jtMsOxkswkYBTwLpAO+M98135gbXo+lwbsoLzr5do+o5RqNFcTzRisZFO5nHOWMWalMWYGsAJrihrVQmTuTKWDnKC1ts8opdzA1UQTC6QbY2xAMdDaYV8KcJ27A1PeE3jgOwACEnX9GaUaauXKlYgIK1eurLXcggULeOuttzwSw8iRIxk5cqRHzl0frnYGOAS0sT/eDwwFVtr/7uHekJQ3HTpeTK+iTZyKiOG8tgneDkepFm/BggWUl5fzs5/9zO3nfuUV31j82NVE8x1WcvkYWATMFpEErDnQbgM+9Eh0qsmtTc9jWEAaZV2v0fYZpXxMSUkJoaGhLpfv16+fB6NxnauXzp4APrM//jPwMtblsqlYSeZB94emvCEzbQPt5SRRfbV9RrVsu3fvZtKkSXTs2JGwsDC6du3KjTfeSHl5+ZkyqampXHHFFYSFhdGlSxeeeeYZZs+efc4qnLm5udxyyy1ERkbSpk0bZsyYwbFjx+qMYeTIkXzzzTd8//33iAgicuZS14IFCxARvv32W2688UbatGnDpZdeCsAPP/zADTfcQHx8POHh4fTu3Zvf/e53FBUVnXN+x0tnlZfzPvzwQ37xi18QHR1Nhw4dmDZtmkvxNpSrU9DsA/bZH5cB/2O/qRZG9le2z+j8ZqplGz9+PG3atOHVV18lOjqa7Oxsli1bRkVFBQB5eXlcddVVdO7cmYULFxISEsILL7xAZmbmOeeaPHkymzdv5plnnqFnz57885//5MEH6/79/corrzBt2jRsNhuvvWbN7hUZWXXFlVtvvZWpU6fyn//850wSPHDgAIMGDeL222+ndevWbN++nblz55Kens4//vGPOp935syZjB8/nvfee49du3bx29/+lsDAQN5+++06j22Ies8M0Fgi0gV4AWsiTsHqtfYrY8yBOo6bA8yuYXeJMSbMoWwA8AhwDxAD7ALmGmMWN/oFtGB5p0roeXojJyJiiWzbzdvhqGbgj+v+yM6CnXUX9KA+7frwyCWP1OuYvLw89uzZw9KlS5kwYcKZ7bfccsuZx/PmzaOwsJDPP/+c+Ph4AMaMGXPOMszLly/nu+++4/333+fmm28+U27s2LFkZWXVGke/fv2IjIykvLycIUOGOC1zww038Kc//anKtilTppx5bIzh8ssvJzIykhkzZvDyyy/Tvn37Wp93+PDh/O///i9gLUm9a9cu5s+ff6YW5W6uXjpzCxGJAL4C+mC17UwHegJfi0irOg6fj9VO5Hi7GqudqHob0ZPAHOAlYCxWt+x/i8g4t7yQFmpdeh6XBqRR1vVyb4eilEe1b9+epKQkHn30Ud544w327NlzTpk1a9YwdOjQM0kGIDw8nOuuq9rJdvXq1QQGBlb58gfOJJ3GmjRp0jnbTpw4wSOPPEL37t0JDQ0lODiY6dOnY4xx+lqqq/4aBg4cSElJCYcPH3ZLzNU1dY3mbiAJ6G2M2QsgIluAPVi1j3k1HWiMyQKq/DwQkelYr+Fth20dgd8Azxljnrdv/lpEegDPAcvc9mpamIwd6xknpyjvO8rboahmor41CV8hIixfvpw5c+Ywa9Ys8vPzSUxM5OGHH+a+++4D4ODBgwwYMOCcYzt16lTl74MHD9K2bVuCg4NrLddQsbGx52y74447WLFiBXPnzmXQoEG0atWKdevW8cADD1BcXFznOasvNV3ZwcCVYxuiSWs0wARgTWWSATDGZADfAxMbcL7bgMPA5w7bxgAhwDvVyr4DDBSRxAY8j18I2L8KgKAkHT+jWr6kpCQWLlxIbm4uGzduZNSoUdx///18+umngPUFf+TIkXOOq/6rPzY2lqNHj1JWVlZruYaqfimruLiYpUuX8vDDDzNz5kxGjBjB4MGDCQ8Pd8vzeUJTJ5r+wDYn27cD9eqHJyLxwJXAu8aYcodd/YESYG+1Q7bb732jv5+POXa6lMRTmzgeFgdtuno7HKWajIgwaNAg5s2zLqhs22Z9RQ0ZMoTVq1dXaWcpKirik08+qXL80KFDsdlsLF5ctQnYlUZ5sGoT1XuL1aakpASbzXZODWrBggUun6OpNfWls3bAUSfbC4C29TzXdKxEWb2bRDvgmDHGVNte4LBfVVPZPlMaP9bboSjlcVu2bGHmzJncdNNN9OjRA5vNxoIFCwgKCmLUKOvS8UMPPcSrr77KmDFjmD17NqGhocybN4/Q0NAqtYzRo0czbNgw7rnnHvLy8s70OqtMWHXp168fr7zyCv/85z/p3r07rVu3pnfv3jWWj4qKYsiQIfzlL38hNjaW6Oho3nrrLbKzsxv3pnhQU9dooOqMz5Ua0s1hBrDRGLPFybnq/Rwi8nMRWS8i63NzcxsQTvOWueMH2sopovpp+4xq+WJiYujatSvz5s1jwoQJTJ06lZycHD7++GMuuugiAKKjo/nyyy9p27YtM2bM4P777+fqq69m0qRJREVFVTlfSkoK48aNY9asWdx0002Ul5fz0ksvuRTLI488wlVXXcVdd93FxRdfzD333FPnMe+//z4XXXQRDzzwALfffjsxMTH89a9/rf8b0UTk3B/+HnwykcPAEmPMPdW2vwLcaIzp4OJ5LgHWYnWL/mu1fX/EmuQz3LFW43DMeGNM1bpvNYMHDzbr1693JZQW440//Ya7T78Bv94OUfF1H6D8SlpaGn379vV2GF5ns9lITk4+k4T8TV2fAxHZYIwZXH17U186247VhlJdP2BHPc5zG1a35vdqeI5QoDtV22kq22bq8zx+4URxGd1OpnKsVTxtNMkodcbjjz9Ojx496NatG/n5+cyfP58tW7awbJl2Xq2Ppk40HwLPi0iSMSYdwD5n2uXAo66cQERCgJuBZcYYZ9e4PgNKgVuxps6pNA3YZu/lphxsyMjnkoCdlMTpJNxKORIR5s6dS05ODiLC+eefz5IlSxg7Vtsy66OpE80bwC+ApSLye6y2lCeBH4HXKguJSDesKW/mGmPmVjvHeKwGfadzJRhjjojIC8AsETkJpAI3YS3U1pAu1C1e5va1XCmFlPa/ytuhKOVT5s6dy9y51b+CVH01aaIxxhSKyCisKWgWYTXQf4nV1nLKoagAgTjvrHAbVg+yj2t5qseAU1htNZVT0PzUGPNRo19EC2QyrfEzId11/IxSyv2afK4z+5xmU+ook0kNvcSMMXXWSuwLtD1lv6lanC4tp9uJDRyNiKdtVJy3w1FKtUDe6N6sfEhqRj4Xy06K4nR+M6WUZ2ii8XMZ29cQKadp11/HzyilPEMTjZ8zGVb7TFiPEV6ORCnVUmmi8WPFZTbij28gP6wrRJ47Q6xSSrmDJho/tml/HoMljaLOl3k7FKVUC6aJxo9lbF1DpBRp+4xSHrBy5UpEhJUrV3o7FK/TROPHTOa3AET00vYZpZTnaKLxU6XlFXQ+toHc0G7QOsbb4SilWjBNNH5q6495XMROiuKGejsUpbxi9+7dTJo0iY4dOxIWFkbXrl258cYbKS8/u45iamoqV1xxBWFhYXTp0oVnnnmG2bNnn7PqZW5uLrfccguRkZG0adOGGTNmcOzYMZdj+eabbxg9ejRRUVG0atWKCy64gDfffBOAcePGnVm6wNHBgwcJCgrixRdfbNDrb0pNPjOA8g0ZW//LRVIE/XR+M+Wfxo8fT5s2bXj11VeJjo4mOzubZcuWUVFRAUBeXh5XXXUVnTt3ZuHChYSEhPDCCy+QmZl5zrkmT57M5s2beeaZZ84sfPbggw+6FMfSpUuZMmUKl19+Oa+99hrR0dFs376d/fv3AzBjxgymTp3Kjh076Nfv7ALB771nTV4/derURr4TnqeJxk/Z0q3xM637jPRuIKpZO/TMM5Sk7fRqDKF9+xDzu9/V65i8vDz27NnD0qVLmTBhwpntt9xyy5nH8+bNo7CwkM8//5z4eGv5jDFjxpCQkFDlXMuXL+e7777j/fff5+abbz5TbuzYsVWWgXbGGMPMmTMZNGgQX3/9NQEB1kWmq6+++kyZiRMnEhkZyaJFi3j22WfPbF+0aBHXXHMNnTp1qtdr9wa9dOaHym0VdD66nsNhCXBeR2+Ho1STa9++PUlJSTz66KO88cYb7Nmz55wya9asYejQoWeSDEB4eDjXXVd1OY3Vq1cTGBjIlClVp3CsTDq12bVrF/v37+euu+46k2SqCw8PZ8qUKbz77rtUruW4detWNm/ezIwZM+p8Dl+gNRo/tD0rnwvZSW7sJG+Hopq5+tYkfIWIsHz5cubMmcOsWbPIz88nMTGRhx9+mPvuuw+w2kAGDBhwzrHVaxAHDx6kbdu2BAcH11rOmfz8fIAqycyZGTNm8Pe//52VK1dy5ZVXsmjRIlq3bs3Eic1j5ROt0fihzC3fc54U6/gZ5deSkpJYuHAhubm5bNy4kVGjRnH//ffz6aefAhAbG8uRI0fOOe7w4cNV/o6NjeXo0aOUlZXVWs6Z6OhoALKzs2stN2LECLp27co777xDRUUF77//PjfccAPh4eF1Pocv0ETjh8rTrfEzUX2u9HIkSnmfiDBo0CDmzZsHwLZt2wAYMmQIq1evrtLOUlRUxCeffFLl+KFDh2Kz2Vi8eHGV7f/4xz/qfO5evXqRkJDA/Pnzz1wWqynGW2+9lf/85z8sW7aMrKysZnPZDPTSmd+xVRhijq7nUGgiMed18HY4SnnFli1bmDlzJjfddBM9evTAZrOxYMECgoKCGDXKquk/9NBDvPrqq4wZM4bZs2cTGhrKvHnzCA0NrdK9efTo0QwbNox77rmHvLy8M73OKhNWbUSEF198kcmTJzNq1CjuvfdeOnToQFpaGkeOHOGJJ86uRj9jxgyeffZZ7r33Xrp06cKIEc1noLXWaPzMrpwCBpmdFHbW8TPKf8XExNC1a1fmzZvHhAkTmDp1Kjk5OXz88cdnxqxER0fz5Zdf0rZtW2bMmMH999/P1VdfzaRJk4iKiqpyvpSUFMaNG8esWbO46aabKC8v56WXXnIplokTJ7J8+XIA7rzzTiZMmMDrr79+Tu+2Pn36MHjwYLKzs5k2bdo5Y3l8mdRWXfNXgwcPNuvXr/d2GB7x8SdLGP/DbeRfN5/2F9/o7XBUM5GWlkbfvn29HYbX2Ww2kpOTzyQhf1PX50BENhhjBlffrpfO/Ez5Pqt9pn0/7QigVF0ef/xxevToQbdu3cjPz2f+/Pls2bKFZcuWeTu0ZkUTjR8xxtCp4AdyQpPo3Kq9t8NRyueJCHPnziUnJwcR4fzzz2fJkiWMHTvW26E1K5po/MjegwVcYHaRHTul7sJKKebOncvcuXO9HUazp50B/Mi+zauIkBKidH4zpVQT0kTjR8r2fUsFQof+On5GKdV0NNH4kfhj68kJTUK0fUY1gPZQ9W+N+ffXNho/Muji4ZSGaZJR9RcSEkJRURERERHeDkV5SVFREaGhoQ06VhONH5ExT9Gwj4nyd9HR0WRlZREdHU3r1q0JCgpqVgMGVcMYYygvL+fkyZPk5eU1eEkCTTRKqTpFRUURGhpKbm4u+fn5VVahVC1bUFDQmRVIw8LCGnYON8eklGqhKpczVqq+tDOAUkopj9JEo5RSyqM00SillPIoTTRKKaU8ShONUkopj9JEo5RSyqN04TMnRCQX2O/tOBxEA3neDqKZ0/ew8fQ9bLyW/h52M8acs0a8JppmQETWO1u1TrlO38PG0/ew8fz1PdRLZ0oppTxKE41SSimP0kTTPLzu7QBaAH0PG0/fw8bzy/dQ22iUUkp5lNZolFJKeZQmGqWUUh6licZHichIETFObse8HZsvEpF4EflfEVktIqft71WCk3JtRWS+iOSJSKGIrBCRgV4I2ee48h6KSEINn0sjIm28E7lvEJEbRGSxiOwXkSIR2SUiz4pI62rl/O4zqInG9/0SGOpwu9q74fisHsBPgaPAKmcFxFoS8kPgWuBBYAoQDHwtIvFNFKcvq/M9dPAsVT+XQ4GTHo3O9/0GsAG/w/qMvQrcBywXkQDw38+gLnzm+9KMMWu8HUQz8K0xphOAiNwFXOOkzARgGDDKGPO1vexqIAP4LVZS92euvIeV0vVzeY7rjTG5Dn9/IyIFwNvASOAr/PQzqDUa1SIYYypcKDYByKn8D24/7jjwETDRU7E1Fy6+h6oG1ZJMpR/s93H2e7/8DGqi8X3viohNRPJF5D0R6ertgJqx/sA2J9u3A11F5Lwmjqc5e1ZEykXkuIh82NLbGBphhP0+zX7vl59BvXTmu44DfwG+AU4AF2Jd+10tIhcaY454M7hmqh2Q6WR7gf2+LXCqyaJpnkqA14AvgFygD9bn8r8icokxJq22g/2JiMQBc4EVxpj19s1++RnUROOjjDEbgY0Om74RkW+BdVjXcX/vlcCaNwGcjVCWpg6kuTLGHATuddi0SkQ+w/pF/hgwzSuB+Rh7zWQpUA7c4bgLP/wMaqJpRowxqSKyG7jY27E0UwVYvyira2u/P9qEsbQYxpgfReQ79HMJgIiEYfUsSwJGGGOyHHb75WdQ22ian5p+Eam6bce6Rl5dP+CAMabFXbJoQvq5BEQkGFgMXAKMM8ZsrVbELz+DmmiaEREZDPQC1no7lmbqQyBORCobaBGRSOB6+z7VAPYOKpfj559L+1iZd4GrgIk1dP/2y8+gXjrzUSLyLlbf+lTgGFZngFlANvC/3ovMd4nIDfaHF9nvx9pXS801xnyD9R95NfCOiDyMdZliFtav8T81dby+qK73UET+gvUDdTVWZ4DeWO9hBfBMU8frY14GbgSeBgpFZIjDviz7JTS//Azq7M0+SkRmAVOBbkAEcAj4FJhtb5BV1YhITR/mb4wxI+1l2gHPAz8BwrD+0z9kjNncFDH6urreQxH5GdZo9x5Aa6xlib8CnjDG7GqiMH2SiGRi/X915gljzBx7Ob/7DGqiUUop5VHaRqOUUsqjNNEopZTyKE00SimlPEoTjVJKKY/SRKOUUsqjNNEopZTyKE00StVARH4mIntEpLRyCW0RyRSRBQ5lRorInMoVFOs4Xxt72WQn+1aKyEo3hu8yEQkQkU0i8j/1OOZC+3LPumyFqpOOo1HKCRHpDBzAmlLkDaDYGLNeRC4EThhj9tnLzQFmA8HGmPI6zpmANdvD3caY+dX29QMwxuxw80upk4jMwBpAmGCMOV2P45YCx4wxt3ksONUi6BQ0SjnXEwgE3jbGfFe50b58g9t5I8E4+A3W63Q5ydi9BiwVkVnGmBwPxKVaCL10plQ19ktjK+1/fikipvJymeOlM4faDECZvZzTSwQOtRmANyrLisjt9v1VLp3ZL8kZEfmJiLwmIgUiclREXhCRQBG5WES+E5FCEdkuImOcPOcIEflSRE7ay30uIgOqlbkUGAi8V217LxH5QESOiEixiBwQkX+LiOOP0y+wFuW7veZ3UylNNEo58yTW4nIADwBD7duqmw+8aX88zF5uaA3nPAhMtj9+1qHsJ3XE8iJQCNwEvAT8yr5tIfCW/ZwFQIqIRFceJCLXAV9irdY4DbgFa26yVSLSxeH81wIngerzbH2Mtc79fcAY4FGs1TXPfGfYLxWutp9DqRrppTOlqjHG7BORyiWJd9Qw3TvGmCwRqVzUam1tbTTGmBIRqbzsll7TOZ34yhjzkP3xcnsC+QVwReUlPRE5iJUorgPetpf9K9ZEmBMrTyQiXwPpwP9gJSyAIcBWY0yFQ7lorEuHE40xjlPXV6n12G0EHhaRAMdzKOVIazRK+bZPq/29Eyh0bDeybwPoAiAiPYHuwLsiElR5A05j1UCGOxzbGWu6f0f5WAnpORG5236+muQCoThfNVIpQBONUr6u+tK+pVjrE51hjCm1Pwyz33e0378JlFW7jQfaOxwehnVJzPF8BhgNrMe6zLdbRNJF5D4n8RXZ78NdeznKH+mlM6Vannz7/SxghZP9pdXKtq1ewBiTDswQEQEuwLpc94qIZBpjHGtZlTWZvEZHrVosrdEo1TiVtQFXftHXp2xj7AIygf7GmPVOblscyu4Ekmo6kbFsAirbiQZUK5II/GiMKUKpGmiNRqnGqRz/8j8i8ilgM8asr6HsYawaxM0isgWrN1mGMSa/hvINYowxIvIA1hiXEOBfWDWOTsBlwAFjzDx78W+BO0SkfWUcInI+VmeCfwJ7scYT3Q6UY62m6ehS+zmUqpHWaJRqnI+BV4D7sRraf6ipoL1X1l1Yl6pW2Mte74mgjDHLsBr9W2F1w/4ca036GHuclZYCxVhtN5UOYc2K8BDWGvfvY3UaGG+M2VBZyN5N+gLgH554Darl0ClolPJz9gGo8caYq+t53CNY42y6G2NsnohNtQyaaJTycyKSCKQBw2q57Ff9mDCsLtCPGmMWejI+1fzppTOl/JwxJgOrDaZjHUUdJWC14yzyQEiqhdEajVJKKY/SGo1SSimP0kSjlFLKozTRKKWU8ihNNEoppTxKE41SSimP+n/uVDPi9iiqOQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(results_lr[\"mean_fit_time\"], results_lr[\"mean_train_score\"], label=\"lr train\")\n",
    "plt.plot(results_lr[\"mean_fit_time\"], results_lr[\"mean_test_score\"], label=\"lr cv\")\n",
    "plt.plot(results_sgd[\"mean_fit_time\"], results_sgd[\"mean_train_score\"], label=\"sgd train\")\n",
    "plt.plot(results_sgd[\"mean_fit_time\"], results_sgd[\"mean_test_score\"], label=\"sgd cv\")\n",
    "plt.xlabel(\"fit time(s)\");\n",
    "plt.ylabel(\"accuracy score\");\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TL;DR if you don't want to wait that long, `SGDClassifier` will probably do better on a big dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # TODO: Can use GridSearchCV to automate this?\n",
    "\n",
    "# n_iter_lr = [10, 30, 100, 300, 1000, 3000]\n",
    "# fit_times_lr = []\n",
    "# train_scores_lr = []\n",
    "# test_scores_lr = []\n",
    "# for n_iter in n_iter_lr:\n",
    "#     lr = LogisticRegression(max_iter=n_iter)\n",
    "#     time = %timeit -o -n1 -r1 lr.fit(X_train, y_train)\n",
    "#     fit_times_lr.append(time)\n",
    "#     train_scores_lr.append(lr.score(X_train, y_train))\n",
    "#     test_scores_lr.append(lr.score(X_test, y_test))\n",
    "    \n",
    "# n_iter_sgd = [100, 300, 1000, 3000, 10000, 30000]\n",
    "# fit_times_sgd = []\n",
    "# train_scores_sgd = []\n",
    "# test_scores_sgd = []\n",
    "# for n_iter in n_iter_sgd:\n",
    "#     lr = SGDClassifier(loss=\"log\", max_iter=n_iter)\n",
    "#     lr.fit(X_train, y_train)\n",
    "#     train_scores_lr.append(lr.score(X_train, y_train))\n",
    "#     test_scores_lr.append(lr.score(X_test, y_test))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
